Papers with memory efficiency
Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) is a key feature for Large Language Models (LLMs) but it faces challenges when dealing with increasing numbers of examples due to performance degradation and quadratic computational costs. |
| Approach: | They propose a Logit Arithmetic Reweighting Approach that uses logit-based ensembling of multiple demonstrations to enhance ICL. |
| Outcome: | The proposed framework outperforms baseline methods in accuracy and memory efficiency. |
FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Unlike other modalities, speech has unique temporal dependencies, making efficient inference methods unexplored. |
| Approach: | They propose a weighted token merging framework specifically designed for speech-related tasks to improve the trade-off between efficiency and performance. |
| Outcome: | The proposed method achieves state-of-the-art efficiency-performance trade-off on speech-related tasks. |
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence. |
| Approach: | They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes. |
| Outcome: | The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models. |
A Hassle-free Algorithm for Strong Differential Privacy in Federated Learning Systems (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Differential privacy (DP) and federated learning (FL) are used for language models training in production mobile keyboard applications. |
| Approach: | They propose a variant of DP-FTRL that uses a correlated noise mechanism to train on-device language models. |
| Outcome: | The proposed method improves privacy-utility trade-off and memory efficiency over existing FL methods while simplifying usage requirements and reducing memory. |
Developing multilingual speech synthesis system for Ojibwe, Mi’kmaq, and Maliseet (2025.naacl-short)
Copied to clipboard
| Challenge: | In general, speech synthesis for Indigenous languages is underdeveloped compared to the majority of languages. |
| Approach: | They propose to train a multilingual model on three typologically similar languages to improve performance over monolingual models. |
| Outcome: | The proposed model can train on three similar languages with high performance and is highly competitive with self-attention architectures with higher memory efficiency. |
See More, Store Less: Memory-Efficient Resolution for Video Moment Retrieval (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing video moment retrieval methods rely on sparse frame sampling, risking information loss. |
| Approach: | a new video-based framework enhances memory efficiency while maintaining high information resolution . SMORE uses query-guided captions to encode semantics aligned with user intent . |
| Outcome: | a new framework improves memory efficiency while maintaining high information resolution . it achieves state-of-the-art performance on QVHighlights, Charades-STA, and ActivityNet-Captions benchmarks . |
EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing key-value (KV) cache compression methods ignore interplays between layer-specific feature patterns and task performance. |
| Approach: | They propose an adaptive framework for layer-wise, task-driven KV cache compression that optimizes memory efficiency and task performance. |
| Outcome: | EvolKV outperforms baseline methods on long-context tasks and surpasses heuristics by 7 percentage points on GSM8K. |
Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to compositional generalization have been designed with semantic parsing in mind. |
| Approach: | They propose a disentangled sequence-to-sequence model which encourages more disentanglement and improves its compute and memory efficiency. |
| Outcome: | The proposed model improves generalization performance across existing tasks and datasets and a new machine translation benchmark. |
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference (2026.findings-eacl)
Copied to clipboard
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae
| Challenge: | Large Language Models (LLMs) are exploding to large sizes, including GPT, LLaMA, and DeepSeek. |
| Approach: | They propose a fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. |
| Outcome: | The proposed method integrates seamlessly with PagedAttention without any modifications to its CUDA attention kernels. |
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer (2022.acl-long)
Copied to clipboard
| Challenge: | Existing sparse attention methods use fixed patterns to select words without considering similarities between words. |
| Approach: | They propose a neural clustering method which integrates into the Self-Attention Mechanism in Transformer and integrates it into the target task. |
| Outcome: | The proposed method outperforms two typical sparse attention methods on translation, text classification, and text matching tasks while having a comparable or even better time and memory efficiency. |
AgentOCR: Reimagining Agent History via Optical Self-Compression (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enabled agentic systems trained with reinforcement learning over multi-turn interaction, but practical deployment is bottlenecked by rapidly growing textual histories that inflate token and memory costs. |
| Approach: | They propose a framework that represents the accumulated observation-action history as a compact rendered image. |
| Outcome: | The proposed framework preserves over 95% of text-based agent performance while significantly reducing token consumption (>50%), yielding consistent token and memory efficiency. |
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (2025.emnlp-main)
Copied to clipboard
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang
| Challenge: | Large foundation models have become huge, but they consume computational resources in pretraining. |
| Approach: | They propose to replace full-size layers with compute-efficient auto-encoders that enforce low-rank activations throughout training. |
| Outcome: | The proposed method reduces the computing cost by 2pmbtimes and improves training throughput by 1.86pmtime. |
QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Structured pruning can reduce model size but results in significant accuracy degradation . quantization and pruning increase the difficulty of fine-tuning, requiring a more refined quantization scheme. |
| Approach: | They propose a structured pruning framework followed by a layer-wise mixed-precision quantization scheme to reduce model memory consumption during fine-tuning and inference. |
| Outcome: | Experiments on benchmark datasets show that QPruner outperforms existing methods in memory savings while maintaining or improving model performance. |
Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Current PP methods face severe bottlenecks, including pipeline bubbles and memory footprint. |
| Approach: | They propose a sequence-level one-forward-one-backward (1F1B) PP method for training LLMs on long sequences with high throughput and memory efficiency. |
| Outcome: | The proposed method achieves 1.14X training throughput with half memory footprint compared to baseline methods . it trains an LLM with 30B parameters on sequences up to 64k tokens using 64X NVIDIA A100 GPUs . |
Forward Knows Efficient Backward Path: Saliency-Guided Memory-Efficient Fine-tuning of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | a number of fine-tuning approaches are available to improve performance of large language models. |
| Approach: | They propose a memory-efficient method to minimize memory associated with cached intermediate activations. |
| Outcome: | The proposed method minimizes memory associated with cached intermediate activations while preserving accuracy. |
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks. however, their extensive memory requirements present significant challenges for deployment in resource-constrained environments. |
| Approach: | They propose a training-free framework that achieves ultra-low equivalent bit-width KV cache quantization. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on TruthfulQA and LongBench. |
Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to compress video inputs ignore the importance of static and dynamic information in long videos, leading to inefficient token usage within limited budgets. |
| Approach: | They propose a token selection strategy that adaptively adjusts static and dynamic information based on question requirements. |
| Outcome: | The proposed method achieves performance improvements (up to 5.8%) on multiple video question answering benchmarks. |
NeuroAda: Activating Each Neuron’s Potential for Parameter-Efficient Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited and require computational and memory resources. |
| Approach: | They propose a parameter-efficient fine-tuning method that enables fine-grained model finetunation while maintaining high memory efficiency. |
| Outcome: | The proposed method reduces CUDA memory usage by up to 60% while maintaining high performance. |
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve instructionfollowing performance of MLLMs often trade off memory efficiency for performance gains, compromising overall efficiency. |
| Approach: | They propose a task-specific expansion and task-general fusion framework based on variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets. |
| Outcome: | The proposed framework improves performance compared to existing benchmarks. |
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. |
| Approach: | They propose an algorithm that uses early LLM layers as filters to select and compress input tokens, reducing the context length for subsequent processing. |
| Outcome: | The proposed method outperforms existing techniques on the Needle in a Haystack task while demonstrating comparable performance on the LongBench challenge. |
AdaLomo: Low-memory Optimization with Adaptive Learning Rate (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models require substantial memory for training, thereby setting a high hardware threshold. |
| Approach: | They propose a low-memory optimization technique that reduces memory footprint . they propose an adaptive learning rate for each parameter and a grouped update normalization to stabilize convergence . |
| Outcome: | The proposed low-memory optimization performs better than the prevailing algorithm for large language models, AdamW. |
Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Quantized training improves computational and memory efficiency but introduces quantization noise. |
| Approach: | They propose to use stochastic rounding to improve LLM training but introduce quantization noise. |
| Outcome: | The proposed method can compensate for reduced accuracy during backpropagation. |
Extracting Prompts by Inverting LLM Outputs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Unlike previous methods, output2prompt only needs outputs of normal user queries. |
| Approach: | They propose a black-box method that extracts the model's prompt without accessing its logits and without adversarial or jailbreaking queries. |
| Outcome: | The proposed method extracts the prompt that generated the outputs without accessing the model's logits and without adversarial or jailbreaking queries. |
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |
KurTail : Kurtosis-based LLM Quantization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Outliers in quantization of large language models make uniform quantization schemes less effective . a new PTQ scheme that leverages Kurtosis-based rotation mitigates outliers . |
| Approach: | They propose a new post-training quantization scheme that leverages Kurtosis-based rotation to mitigate outliers in the activations of large language models. |
| Outcome: | The proposed method outperforms existing quantization methods with 13.3% boost in MMLU accuracy and 15.5% boost in Wiki perplexity. |
Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression (2026.acl-long)
Copied to clipboard
Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin
| Challenge: | Recent methods to reduce the KV cache size fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss. |
| Approach: | They propose an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant. |
| Outcome: | The proposed method can maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting. |
Reinforcement Learning for Diffusion LLMs via Energy-Based Gibbs Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Diffusion Large Language Models (dLLMs) offer parallel decoding and bidirectional context modeling . aligning dLLms with reinforcement learning (RL) remains a challenge . |
| Approach: | They propose a variational framework that reformulates RL for dLLMs as a distribution matching problem. |
| Outcome: | The proposed framework reformulates RL for dLLMs as a distribution matching problem. |